Distributed offline indexing

ABSTRACT

A distributed offline indexing system uses a set of data processing systems in a distributed computing environment to create an index database that can store, for example, data about geographical or geospatial areas. The index database can be distributed across a plurality of database shards such that each shard includes an index file and a DB file. The index files include keys that refer to values in their corresponding DB files. The keys are used to look-up their corresponding values at search time. At indexing time, the keys are hashed, with an entropy creating hash, to distribute the keys across the shards.

This application claims priority to and the benefit of U.S. ProvisionalPatent Application No. 62/323,507, filed Apr. 15, 2016, whichapplication is hereby incorporated herein by reference.

BACKGROUND

The embodiments described herein relate to distributed computingenvironments and to processes in those environments that createdatabases.

The creation of large databases can require enormous computingresources. For example, a database that provides information aboutbusinesses (e.g. coffee shops, grocery stores, restaurants, etc.) in aspecific geographic or geospatial region (such as a zip code or areadefined by one or more pairs of a latitude and a longitude) can requiremany hours to create even if the creation process uses a distributedcomputing environment (e.g. online distributed indexing) in whichparallel processing is performed concurrently with multiple dataprocessing systems. As the geographical data in a region can changefrequently over time, it is often necessary to create a new databasefrequently. Moreover, a geographic or geospatial database can often bedefined by “keys” that subdivide the “map” into very small regions, andthus the number of keys can become very large. For example, databasescan use a design based on key, value pairs, where the key defines theregion and the value is the information returned as a result of a searchusing the key as the search query. In the case of Bluedot®, which is aservice (from Bluedot® Innovation) that provides contextual informationto devices without resorting to GPS to provide position (e.g.latitude/longitude), the number of keys is over a billion. Even withmany data processing systems in a distributed computing environment, itcan take many hours to create a database that contains over a billionkeys. The present description provides embodiments that can improvethese creation times.

SUMMARY OF THE DESCRIPTION

The embodiments described herein can operate in a distributed computingenvironment that creates a partitioned database having a plurality ofshards, each shard being one of the partitions, and each shard can be adense index that includes both a key file (that includes a set of keys)and a database file (that includes the corresponding set of values) thatcan be implemented as an append only file in one embodiment. Thedistributed computing environment can hash the keys with a hash functionin order to generate a partition number or identifier to allocate theindexing of the key, value pairs to the various partitions on a somewhateven basis to distribute the indexing workload evenly. In oneembodiment, the hash function is configured to provide entropy acrossthe created hash values for the different keys to spread the keys amongthe different shards such that keys for a dense geospatial area (e.g.New York City) will be spread across most if not all shards/partitions.

In one embodiment, a distributed computing environment can perform amethod comprising the following operations: receiving a plurality ofinputs, each input comprising a key and a value pair; computing, foreach key, a hash value using a hash function configured to provideentropy across hash values for different keys to distribute the keysacross a plurality of partitions; determining, based on each hash value,a partition identifier and mapping the corresponding key (that washashed to create the hash value) to the determined partition identifier;sorting, separately within each partition on a data processing systemdedicated to processing the partition, keys mapped to the partition toprovide a sorted set of keys mapped to the partition; storing, withineach partition, the keys and their paired values, the keys being storedin an index file and their paired values being stored in a databasefile, wherein each key refers to its paired value in the database filewithin the partition. In one embodiment, each partition, in theplurality of partitions, is a database shard, and the keys representdifferent geographic or geospatial areas (e.g. zip code or regiondefined by latitude/longitude, etc.) and one city or geographic regioncan be dispersed across all shards due to the entropy. In oneembodiment, the entropy tends to balance the indexing workload (e.g. thesorting and the writing/storing of the keys and the values/databasefiles) and the storage usage across all of the data processing systemsthat create and store all of the shards, and the entropy disperses thekeys for one geographic region (e.g. New York City) across a plurality(e.g., all) shards. In one embodiment, the keys are used to search forand retrieve values by hashing each of the keys (e.g., a search query orrequest specifies directly or indirectly a key which is then hashed withthe same hash function used in the process of creating the hash valuethat was used to identify a partition for the key); the hashing of thekey (specified by the search query) identifies the partition/shardcontaining the key, and then the key is used to find the correspondingvalue within the identified partition/shard.

In one embodiment, each partition or shard has a set of one or more dataprocessing systems dedicated to sorting and storing the keys and theirvalues within the identified partition; for example, a partitionersystem can define the partitions by computing hash values for all inputkeys and then can dispatch for dedicated processing all key, valuepairs, for a particular partition/shard to a particular set of systemsthat are dedicated to sorting and storing the keys (and their values) inthat particular partition/shard.

In one embodiment, each partition or shard includes a Bloom filter thatstores values representing geospatial tiles in a two-dimensional map.The values stored in the Bloom filter can be quadtree keys representingtiles in a quadtree layout that represents a geospatial area. In oneembodiment, the quadtree keys are represented in base 10 values with ashift offset to provide unicity across all nodes in a quadtree layout,wherein each node in the quadtree contains one of the quadtree keys.

The methods described herein can be implemented by a variety ofdifferent data processing systems with different architectures. Themethods described herein can also be implemented by one or more dataprocessing systems which execute executable instructions, stored on oneor more non-transitory machine readable media, that cause the one ormore data processing systems to perform the one or more methodsdescribed herein. Thus the embodiments described herein include methods,data processing systems, and non-transitory machine readable media.

The above summary does not include an exhaustive list of all embodimentsin this disclosure. All systems and methods can be practiced from allsuitable combinations of the various aspects and embodiments summarizedabove, and also those disclosed in the Detailed Description below.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is illustrated by way of example and notlimitation in the figures of the accompanying drawings in which likereferences indicate similar elements.

FIG. 1 shows an example of a distributed computing environment accordingto one embodiment.

FIG. 2 shows an example of an input or set of inputs which can beprovided to a distributed computing environment.

FIG. 3A shows an example of a data store containing a plurality ofshards.

FIG. 3B provides a more detailed illustration of a particular shardwhich contains an index file and a database file.

FIG. 3C shows an example of a data store which includes shards, each ofwhich includes a Bloom filter used in an alternative embodiment.

FIG. 4 is a flowchart which illustrates a method according to one ormore embodiments described herein.

FIG. 5 illustrates a search process using a data store, such as the datastore 301 in one embodiment.

FIG. 6A shows a subdivided geographical region, such as a map which hasbeen subdivided into four quadrants.

FIG. 6B shows the same geographical region of FIG. 6A except that two ofthe quadrants have been further subdivided into sub-quadrants.

FIG. 6C shows a quadtree representation of the sub-divided geographicalregion 601.

FIG. 7 shows an example of a data processing system which can be used toimplement one or more embodiments described herein.

DETAILED DESCRIPTION

Various embodiments and aspects will be described with reference todetails discussed below, and the accompanying drawings will illustratethe various embodiments. The following description and drawings areillustrative and are not to be construed as limiting. Numerous specificdetails are described to provide a thorough understanding of variousembodiments. However, in certain instances, well-known or conventionaldetails are not described in order to provide a concise discussion ofembodiments.

Reference in the specification to “one embodiment” or “an embodiment”means that a particular feature, structure, or characteristic describedin conjunction with the embodiment can be included in at least oneembodiment. The appearances of the phrase “in one embodiment” in variousplaces in the specification do not necessarily all refer to the sameembodiment. The processes depicted in the figures that follow areperformed by processing logic that comprises hardware (e.g. circuitry,dedicated logic, etc.), software, or a combination of both. Although theprocesses are described below in terms of some sequential operations, itshould be appreciated that some of the operations described may beperformed in a different order. Moreover, some operations may beperformed in parallel rather than sequentially.

FIG. 1 shows an example of a distributed computing environment which cancontain a plurality of data processing systems, such as desktop computersystems or server systems each of which is controlled, in oneembodiment, by software that enables a distributed computing environmentsuch that the different computers in the environment are coordinated towork together and perform specific tasks as specified by one or moresystems which manage the environment to ensure that the data processingtasks are distributed among the various data processing systems to allowfor concurrent and parallel data processing as is known in the art. Inthe environment 101 shown in FIG. 1, a plurality of mapper systems, suchas mapper systems 104A through 104N receive an input 102 and processthat input by determining or parsing the various portions of the inputinto different categories. For example in one embodiment, the input maybe formatted in a conventional key, value pair, and each input can be inthis key, value pair format.

FIG. 2 shows an example of a set of key, value pairs in one embodiment.For example, the input 102 shown in FIG. 2 shows that each key canspecify a zip code or other geographic region (by for example,specifying a latitude and a longitude or a set of latitude andlongitudes), and for each key, there is a corresponding set ofinformation or values, such as a list of businesses within theparticular zip code, etc. In one embodiment, a geographic region, suchas a city or a state or a nation can be sub-divided into regions, eachregion being specified by a key, and then corresponding values areassigned to that key. Each of the mapper systems 104A through 104Nprocesses the inputs to transform or parse the inputs so that the keyscan be recognized and processed by the partitioner system 105.

In the embodiment shown in FIG. 1, the partitioner system 105 can be asingle data processing system or a set of data processing systems in analternative embodiment. The partitioner system 105 receives theplurality of keys from the mapper systems 104A through 104N andprocesses these keys in a manner described below in conjunction withFIG. 4 to derive a partition identifier or number for each key based ona computation performed by the one or more partitioner systems 105.

The result of the operations of the partitioner system 105 provides aset of keys for each partition, where each key has now been assigned (bythe partitioner system 105) to a particular partition based upon thenumber of partitions in the computing environment 101. In the exampleshown in FIG. 1, there are N partitions or N shards 108A through 108N.In one embodiment, there may be 10 or 15 shards which disperse the keysacross the shards as will be described further below.

The partitioner system or set of partitioner systems 105 provide thekeys and their corresponding values along with the identified partitionfor each key to a set of sorter systems 106A through 106N. Each of thesesorter systems can be dedicated to processing only those keys assignedto a particular shard in one embodiment. Thus, each sorter system inthis embodiment can be dedicated to processing keys and theircorresponding values within only a particular partition. This has theadvantage that it distributes the sorting work essentially orsubstantially evenly across the sorter systems if the partitioner canallocate the keys substantially evenly across the N shards. The sortersystems each sort the keys in an order (e.g., sorting lexicographically)to provide a sorted list of keys which then can be stored in an indexfile within the shard according to one embodiment. The output from eachsorter system is coupled to one or more reducer systems, such as reducersystems 107A through 107N. The reducer systems receive the key and valuepairs which have been sorted and create the shards by storing the keysand the values for its corresponding shard. In one embodiment, eachreducer system is dedicated to creating only a particular shard suchthat each reducer system creates only the shards required to be createdfor its assigned shard. In this way, the workload of creating the shardsamong the reducer systems is distributed substantially evenly across theshards if the partitioner system 105 is able to create an evendistribution of keys across the N shards.

The output from each reducer system, such as reducer system 107A, isstored on the corresponding partition or shard. In the case of a reducersystem 107A, it stores its output in the shard 108A while the reducersystem 107N stores its output in shard 108N.

FIG. 3A shows an example of a data store 301 which includes the Nshards. In the embodiment shown in FIG. 3A, each shard includes a denseindex which includes an index file as well as a database (DB) file whichcontains the values corresponding to the keys stored in the index filewithin the same shard. For example, index file 307A includes a pluralityof keys while database file 307B includes the corresponding values forthose keys in the index file 307A. Each of the shards in the data store301, such as shards 302, 303, 304, and 305 each include an index fileand a database file. For example, shard N, which is shard 305, includesan index file 311A and a database file 311B. The data store 301represents the set of N shards shown in FIG. 1 after they have beencreated by the distributed computing environment 101 in one embodiment.

FIG. 3B shows a more detailed example of one particular shard. It willbe appreciated that each shard within a data store can be similar to theshard 302 shown in FIG. 3B. A shard 302 includes the index file 307A andthe database file 307B. Within the index file 307A, there is a headerand set of sorted keys, such as key 315 and a corresponding offset, suchas offset 316. Each offset is computed based upon, in one embodiment,the corresponding payload value which in the case of offset 316 is thepayload 319 which is the value associated or corresponding to key 315.Similarly, the value corresponding to key 2 in index file 307A ispayload 2. The size of payload 319 is indicated by the size data 318within the database file 307B. In one embodiment, the database file 307Bcan be implemented as an append only file in which new content or valuesare added at the end of the file. The keys, such as key 315 are sortedin one embodiment lexicographically by one of the sorter systems in adistributed computing environment as described herein. The keys arestored in the sorted order to allow searching using techniques known inthe art for searching a sorted list of keys. Further information aboutsearching will be provided below in conjunction with a description ofFIG. 5.

FIG. 3C shows another example of a data store containing a plurality ofshards according to an alternative embodiment. The data store 351includes N shards, such as shard 353 and shard 354. Each shard includesa Bloom filter, an index file, and a database file. In particular, shard353 includes a Bloom filter 356, an index file 357, and a database file358. Similarly, shard 354 includes a Bloom filter 359, an index file360, and a database file 361. The index files and the database fileswithin the shards shown in FIG. 3C can be similar to the index files andthe database files shown in FIG. 3B. Each of the shards shown in FIG. 3Calso includes a Bloom filter for that particular shard. The Bloom filteris used to store, in one embodiment, quadtree keys representing tiles ina quadtree layout that represents a geographical or geospatial area suchas a map of a city or town or state, etc. In one embodiment, thequadtree keys can be represented in a base 10 value with a shift offsetto provide unicity across all modes in the quadtree layout. The Bloomfilter allows a quick search to determine whether any data is presentfor a particular region. It is often the case that certain geographicalregions are very sparse or so sparse that they contain no values. Ageographic region such as a desert or tundra may contain no values forexample. Thus, rather than searching through or using other techniquesfor finding a key in the sorted list of keys within the index file, thekey for a region can be provided as an input to the Bloom filter whichcan then indicate whether or not any values are stored for that key.This can be implemented by not storing the quadtree key for a tile orregion when there are no values for that key. In other words, if a tileor region is empty of values then the quadtree key for that tile is notstored in the Bloom filter. Thus when the Bloom filter is searched usingthat quadtree key, the Bloom filter will report that there are nocorresponding values for that key and the search can end at that pointwithout using techniques known in the art to search the sorted list ofkeys within the index file.

In one embodiment, each key is assigned to only one shard by thepartitioner system 105. Moreover, in one embodiment, the partitionersystem 105 attempts to distribute the keys substantially evenly acrossthe shards, and this can mean that all sorters and all reducers in thepipeline of the distributed computing environment 101, for example, willget a similar, uniformly distributed amount of keys to index. Further,in one embodiment, the partitioner uses a hash function which isconfigured to provide entropy across hash values produced by the hashfunction for different keys in order to distribute the keyssubstantially evenly across the partitions or shards. Moreover, theentropy provided by the hash function can attenuate a hot spot, such asNew York City or other geographical regions where there may be many,many values for a particular key. Thus, the hash function, by providingentropy, can distribute keys for such a particular geographic region,such as New York City, across most if not all of the partitions orshards rather than concentrating that particular geographic regionwithin a particular shard.

FIG. 4 shows a method which can be performed using the distributedcomputing environment shown in FIG. 1. In operation 401, the distributedcomputing environment receives the inputs and extracts the keys. Theextraction of the keys can be performed in one embodiment by a set ofmapper systems, such as mapper systems 104A through 104N. Then inoperation 403, a partitioner, such as partitioner 105 computes a hash ofeach key. In one embodiment, the hash is performed by using the hashfunction which is configured to provide entropy across the produced hashvalues for all of the different keys to thereby distribute the keyssubstantially evenly across all of the shards or partitions beingcreated by the distributed computing environment. In one embodiment, thehash function can be the hash function known as the SHA-1 hash. Then inoperation 405 the partitioner can perform optional operations which areconfigured to provide a particular identifier or number of a partitionfor the key currently being processed. In one embodiment, a modulo Noperation can be performed in order to obtain the final partition numberfor the particular key being currently processed. For example, in oneembodiment, the number of partitions can be set at N (e.g. N=10), andthe number N can be used in a module N operation to derive the shardidentifier (shard_id) for each key. For example, the following functioncan be used to compute the shard identifier (where the hashed key willbe interpreted as a 64 bit integer):shard_id=hash(key) % N_Partition

In addition, operation 405 can also include an optional operation inwhich the number of keys in each partition is counted. This can beperformed by keeping a running sum for each partition of the keys thathave been allocated to a particular partition. The count can be used toverify that the keys are sufficiently dispersed across the shards.

Then in operation 407, each key is assigned to a particular partition byassigning a partition number or partition identifier to each key basedupon the hash value computed in operation 403. Then in operation 409,the keys are sorted within each partition. In one embodiment, each ofthe sorters in a distributed computing environment, such as sorters 106Athrough 106N performs this sorting operation for its particularpartition to which it is dedicated. Then in operation 411, the reducers,such as reducer systems 107A through 107N, create the data for eachshard and store that data, such as a dense index, for each shard. Thereducers in one embodiment also can create the quadtrees and quadtreekeys and Bloom filters for each partition.

The method shown in FIG. 4 is repeated for each key, and each key alongwith its corresponding value is stored to create, for example, the denseindex stored within each shard which ultimately creates the data storethat can be searched, such as data store 301 shown in FIG. 3A.

FIG. 5 will now be described to provide an example of how the data storecan be searched at run time after the data store has been created. Thedata store can receive an input to perform a search, such as a searchquery which is shown as get 503 in FIG. 5. This in turn causes a keydispatcher 504 in the process 501 to perform a hash function on theinput which in this case is the zip code 95014. The hash function 505returns a hash value which can then be used to determine the particularshard in which the key is stored within the index file 508 in the shard507 in the data store 506. The hash function 505 is in one embodimentthe same hash function which was used in operation 403 to assign the keyto the shard 507. Further, any operations (e.g. modulo N) that wereperformed to derive the partition identifier for that key are alsoperformed with the hash function 505 in order to derive the particularshard that stores the key in the search query. Using techniques that areknown in the art, the key can then be used to search the index file 508to then cause the retrieval of the value corresponding to the key“95014” from DB file 510 to return the result 511 which in this case isthe city Cupertino.

Another aspect of the embodiments described herein relate to the use ofbase 10 quadtree keys stored within nodes of a quadtree. Quadtrees havebeen used in the past to store a locational code derived from the Zorder of the quadrants of a geographical region, such as a map.

FIG. 6A shows an example of a geographical region 601, which can be amap of a region which has been subdivided into four quadrants which arequadrants 602, 603, 604, and 605. The numbers 0, 1, 2, and 3 have beenassigned to each of the respective quadrants according to the Mortoncode or Z-order code, and this is repeated when the quadrants aresubdivided to create sub-quadrants as shown in FIG. 6B which shows eightsub-quadrants in the upper half of the geographic region 601 which hasnow become geographic region 601A after the further subdivision. Thesub-quadrants include sub-quadrants 607, 608, 609, and 610 which are allcontained within the quadrant 602. The values shown within eachsub-quadrant can be the quad key values, each of which is the address ofthe tile represented by the quadtree key or quadkey. For example,quadkey “00” is the address for the sub-quadrant 607 and is representedby a number in base 4 values.

FIG. 6C shows an example of a quadtree containing the quadtree keyswhich specify the address of the tiles shown in FIG. 6B. Thus, the node607A includes a quadtree key “00” (in base 4) which corresponds to thesub-quadrant 607 shown in FIG. 6B. Similarly, the node 608A includesquadtree key “01” (in base 4) which corresponds to the sub-quadrant 608shown in FIG. 6B, the node 609A includes quadtree key “02” (in base 4)which corresponds to the sub-quadrant 609 shown in FIG. 6B, and the node610A includes quadtree key “03” (in base 4) which corresponds to thesub-quadrant 610 shown in FIG. 6B. Similarly, node 602A specifies theaddress for the quadrant 602, and nodes 603A, 604A, and 605Arespectively specify the addresses of the quadrant 603,604, and 605. Asis known in the art, the quadtrees can be indexed by using the quadkeyas the address of the tile in order to retrieve the content from thattile at search time. The methods in the prior art use base 4 addressvalues as the tile keys. In one embodiment described herein, base 10values rather than base 4 values are used in order to significantly savespace by representing the sequence in a 64 bit integer instead of astring that would otherwise be equal to the zoom level in bytes. Inorder to do this, in one embodiment, a shift key that allows thedefinition of a unique key for both leaf and internal nodes is used. Theshift key represents an offset in the hierarchy from which a givenlocational code will be defined with. This offset allows the use of acompact representation in base 10 of each quadkey while maintainingunicity. It is defined such that:

$\frac{K( {{lat},{lng},z} )}{{Shift}\mspace{14mu}{Key}} = {( \ {{\frac{\sum\limits_{n = 1}^{n = {z - 1}}}{{Shift}\mspace{14mu}{O{ffset}}}\ 4^{n}} - 1} ) + \frac{Q{K_{10}( {{lat},{lng},z} )}}{{Quadkey}\mspace{14mu}{Base}\mspace{14mu} 10}}$

Where:

-   -   Z represents the zoom level    -   Lat and lng respectively represents the latitude and longitude        as defined in the WGS84 datum.    -   The sum operation here defines a recursive computation where the        current value is defined by the previous value until reaching        the lower bound (1 in this case) or the upper bound (z−1). “n”        is a common named variable used in the sum notation as a holder        of the current iteration in this case, “n” is a varying integer        that is sequentially assigned a value in between [1, Z−1].

FIG. 7 shows one example of a data processing system which may be usedwith any one of the embodiments described herein as one of the dataprocessing systems in a distributed computing environment. Note thatwhile FIG. 7 illustrates various components of a data processing system,it is not intended to represent any particular architecture or manner ofinterconnecting the components as such details are not germane to thisdescription. The system shown in FIG. 7 can be used to implement each ofthe systems in a distributed computing environment such as partitioner105 or sorter 106N, etc.

As shown in FIG. 7, the computer system 800, which is a form of a dataprocessing system, includes a bus 803 which is coupled to one or moremicroprocessor(s) 805 and a ROM (Read Only Memory) 807 and volatile RAM809 (e.g. DRAM) and a non-volatile memory 811. The one or moremicroprocessors 805 are coupled to optional cache 804. The one or moremicroprocessors 805 may retrieve the stored instructions from one ormore of the non-transitory memories 807, 809 and 811 and execute theinstructions to perform operations described above. These memoriesrepresent examples of machine readable non-transitory storage media thatcan store or contain computer program instructions which when executedcause a data processing system to perform the one or more methodsdescribed herein. The bus 803 interconnects these various componentstogether and also interconnects these components 805, 807, 809 and 811to a display controller and display device 813 and to peripheral devicessuch as input/output (I/O) devices 815 which may be one or more of mice,touch screens, touch pads, touch sensitive input devices, keyboards,modems, network interfaces, printers and other devices which are wellknown in the art. Typically, the input/output devices 815 are coupled tothe system through input/output controllers 817. The volatile RAM(Random Access Memory) 809 is typically implemented as dynamic RAM(DRAM) which requires power continually in order to refresh or maintainthe data in the memory.

The mass storage 811 is typically a magnetic hard drive or a magneticoptical drive or an optical drive or a DVD RAM or a flash memory orother types of memory system which maintain data (e.g., large amounts ofdata) even after power is removed from the system. Mass storage 811 is apersistent memory. Typically the mass storage 811 will also be a randomaccess memory although this is not required. While FIG. 7 shows that themass storage 811 is a local device coupled directly to the rest of thecomponents in the data processing system, it will be appreciated thatone or more embodiments may utilize a non-volatile memory which isremote from the system, such as a network storage device which iscoupled to the data processing system through a network interface suchas a modem, an Ethernet interface or a wireless network. The bus 803 mayinclude one or more buses connected to each other through variousbridges, controllers and/or adapters as is well known in the art.

In the foregoing specification, specific exemplary embodiments have beendescribed. It will be evident that various modifications may be made tothose embodiments without departing from the broader spirit and scopeset forth in the following claims. The specification and drawings are,accordingly, to be regarded in an illustrative sense rather than arestrictive sense.

What is claimed is:
 1. A non-transitory machine readable medium storingexecutable instructions which when executed by one or more dataprocessing systems that implement a partitioned database in adistributed computing environment, cause the one or more data processingsystems to perform a method comprising: receiving a plurality of inputs,each input comprising a key and value pair to be stored in a partitionof a plurality, N, of partitions of the partitioned database, where ineach partition comprises a database shard having a Bloom filter thatstores values representing geospatial tiles or regions in a map;computing, for each key, a single hash value using a hash functionmodulo N, the hash function configured to provide entropy across hashvalues for different keys to distribute the keys substantially evenlyacross the plurality of N partitions; determining, based on the hashvalue of each key, modulo N, a partition identifier and mapping thecorresponding key to the determined partition identifier; for eachpartition identifier having one or more keys mapped to the partitionhaving the partition identifier: distributing the one or more keys andtheir paired values to the partition having the partition identifier;performing, separately within each partition on a data processing systemdedicated to processing the partition; sorting the one or more keysdistributed to the partition into an ordering to provide a sorted set ofkeys mapped to the partition; storing the sorted keys into an index fileand storing their paired values in a database file, wherein each keyrefers to its paired value in the database file within the partition;wherein the processing operations for the partition further includeprocessing one or more queries directed to one or more keys that, whenhashed modulo N, map to the partition; wherein a key and the valueassociated with the key are accessed by hashing the key modulo N todetermine the partition associated with the key and sending the key andan access request to the determined partition; and in response todetermining that the key is not present in a tile of the Bloom filterfor the partition, determining that the key is not present in thepartition, such that no central index or central mapping of keys topartitions is maintained in the distributed computing system.
 2. Themedium as in claim 1 wherein each database shard is an append-only file.3. The medium as in claim 1 wherein the keys represent differentgeospatial areas and keys for at least one city or geospatial region aredispersed across most, or all shards, due to the entropy.
 4. The mediumas in claim 3 wherein the entropy balances an indexing workload andstorage usage across all of the data processing systems that process allof the shards and wherein the entropy disperses keys for one geospatialarea across a plurality of shards.
 5. The medium as in claim 1 furthercomprising: receiving a query against the partitioned database;determining one or more keys derived from the query; for each key of theone or more keys: determining a partition identifier corresponding tothe key by hashing the key, modulo N; transmitting the key to thepartition having the partition identifier to query the partition for avalue corresponding to the key; and returning a value corresponding tothe key, in response to determining that the key matches a key in theindex file stored in the partition.
 6. The medium as in claim 1 whereineach partition has a set of one or more data processing systemsdedicated to sorting and storing within the partition and whereindetermining partition identifiers is performed by a single dataprocessing system.
 7. The medium as in claim 1 wherein accessing a valuehaving a key in a determined partition includes: determining whether thekey is present in a tile or region of a Bloom filter for the partition,and in response to the key being present in the tile or region of theBloom filter, searching the index file of the partition for the key, andreturning the value corresponding to the key, otherwise: returning anull value.
 8. The medium as in claim 1 wherein the values stored in theBloom filter are quadtree keys representing tiles in a quadtree layoutthat represents a geospatial area.
 9. The medium as in claim 8 whereinthe quadtree keys are represented in base 10 values with a shift offsetto provide unicity across all nodes in the quadtree layout.
 10. Themedium of claim 1, wherein the hash function is a secure hash function(“SHA”).
 11. The medium of claim 1, wherein generating and indexing thepartitioned database are performed offline.
 12. A method practiced oneor more data processing systems that implement a partitioned database ina distributed computing environment, the method comprising: receiving aplurality of inputs, each input comprising a key and value pair to bestored in a partition of a plurality, N, of partitions of thepartitioned database, wherein each partition comprises a database shardhaving a Bloom filter that stores values representing geospatial tilesor regions in a map; computing, for each key, a single hash value usinga hash function, modulo N, the hash function configured to provideentropy across hash values for different keys to distribute the keyssubstantially evenly across the plurality of N partitions; determining,based on the hash value of each key modulo N, a partition identifier andmapping the corresponding key to the determined partition identifier;for each partition identifier having one or more keys mapped to thepartition identifier: distributing the one or more keys and their pairedvalues to the partition having the partition identifier; performing,separately within each partition on a data processing system dedicatedto processing the partition; sorting the one or more keys distributed tothe partition into an ordering to provide a sorted set of keys mapped tothe partition; storing the sorted keys into an index file and storingtheir paired values, in database file in the partition, wherein each keyrefers to its paired value in the database file within the partition;wherein the processing operations for the partition further includeprocessing one or more queries directed to one or more keys that, whenhashed modulo N, map to the partition; wherein a key and the valueassociated with the key are accessed by hashing the key modulo N todetermine the partition associated with the key and sending the key andan access request to the determined partition; and in response todetermining that the key is not present in a tile of the Bloom filterfor the partition, determining that the key is not present in thepartition, and no central index or mapping of keys to partitions ismaintained in the distributed computing system.
 13. The method as inclaim 12 wherein each database shard is an append-only file.
 14. Themethod as in claim 12 wherein the keys represent different geospatialareas and keys for at least one city or geospatial region are dispersedacross most, or all, shards due to the entropy.
 15. The method as inclaim 14 wherein the entropy balances an indexing workload and storageusage across all of the data processing systems that process all of theshards and wherein the entropy disperses keys for one geospatial areaacross a plurality of shards.
 16. The method as in claim 12 furthercomprising: receiving a query against the partitioned database;determining one or more keys derived from the query; for each key of theone or more keys: determining a partition identifier corresponding tothe key by hashing the key, modulo N; transmitting the key to thepartition having the partition identifier to query the partition for avalue corresponding to the key; and returning a value corresponding tothe key, in response to determining that the key matches a key in theindex file for the partition stored in the partition.
 17. The method asin claim 12 wherein each partition has a set of one or more dataprocessing systems dedicated to sorting and storing within the partitionand wherein determining partition identifiers is performed by a singledata processing system.
 18. The method as in claim 12 wherein accessinga value having a key in a determined partition includes: determiningwhether the key is present in a tile or region of a Bloom filter for thepartition, and in response to the key being present in the tile orregion of the Bloom filter, searching the index file of the partitionfor the key, and returning the value corresponding to the key,otherwise: returning a null value.
 19. The method as in claim 12 whereinthe values stored in the Bloom filter are quadtree keys representingtiles in a quadtree layout that represents a geospatial area.
 20. Themethod as in claim 19 wherein the quadtree keys are represented in base10 values with a shift offset to provide unicity across all nodes in thequadtree layout.